notebook.community

Edit and run

Analysis of two oyster samples where Lotterhos did methylRAD

The M2 and M3 samples are here:

http://owl.fish.washington.edu/nightingales/C_gigas/9_GATCAG_L001_R1_001.fastq.gz http://owl.fish.washington.edu/nightingales/C_gigas/10_TAGCTT_L001_R1_001.fastq.gz





In [40]:



    


bsmaploc="/Applications/bioinfo/BSMAP/bsmap-2.74/"

Genome version





In [12]:



    


!curl \
ftp://ftp.ensemblgenomes.org/pub/release-32/metazoa/fasta/crassostrea_gigas/dna/Crassostrea_gigas.GCA_000297895.1.dna_sm.toplevel.fa.gz \
> /Volumes/caviar/wd/data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa.gz



    


















    






  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  148M  100  148M    0     0  5192k      0  0:00:29  0:00:29 --:--:-- 5790k

















In [7]:



    


!curl ftp://ftp.ensemblgenomes.org/pub/release-32/metazoa/fasta/crassostrea_gigas/dna/CHECKSUMS



    


















    






08778 148199 Crassostrea_gigas.GCA_000297895.1.dna.nonchromosomal.fa.gz
08778 148199 Crassostrea_gigas.GCA_000297895.1.dna.toplevel.fa.gz
57175 143732 Crassostrea_gigas.GCA_000297895.1.dna_rm.nonchromosomal.fa.gz
57175 143732 Crassostrea_gigas.GCA_000297895.1.dna_rm.toplevel.fa.gz
45604 151782 Crassostrea_gigas.GCA_000297895.1.dna_sm.nonchromosomal.fa.gz
45604 151782 Crassostrea_gigas.GCA_000297895.1.dna_sm.toplevel.fa.gz
62118     5 README

















In [13]:



    


!ls /Volumes/caviar/wd/data/



    


















    






Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa.gz

















In [14]:



    


!md5 /Volumes/caviar/wd/data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa.gz



    


















    






MD5 (/Volumes/caviar/wd/data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa.gz) = c70084d76bd6d7a1ba52c13843e69ccc

















In [17]:



    


cd /Volumes/caviar/wd/



    


















    






/Volumes/caviar/wd

















In [22]:



    


mkdir $(date +%F)



    











In [24]:



    


ls



    


















    






2016-10-11/ data/

















In [ ]:



    


ls /Volumes/web/nightingales/C



    











In [28]:



    


!curl \
http://owl.fish.washington.edu/nightingales/C_gigas/9_GATCAG_L001_R1_001.fastq.gz \
> /Volumes/caviar/wd/2016-10-11/9_GATCAG_L001_R1_001.fastq.gz



    


















    






  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  560M  100  560M    0     0  55.6M      0  0:00:10  0:00:10 --:--:-- 77.8M

















In [32]:



    


!curl \
http://owl.fish.washington.edu/nightingales/C_gigas/10_TAGCTT_L001_R1_001.fastq.gz \
> /Volumes/caviar/wd/2016-10-11/10_TAGCTT_L001_R1_001.fastq.gz



    


















    






  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  619M  100  619M    0     0  46.1M      0  0:00:13  0:00:13 --:--:-- 44.0M

















In [36]:



    


cd 2016-10-11/



    


















    






/Volumes/caviar/wd/2016-10-11

















In [38]:



    


!cp 9_GATCAG_L001_R1_001.fastq.gz M2.fastq.gz



    











In [39]:



    


!cp 10_TAGCTT_L001_R1_001.fastq.gz M3.fastq.gz



    











In [41]:



    


for i in ("M2","M3"):
    !{bsmaploc}bsmap \
-a {i}.fastq.gz \
-d ../data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa \
-o bsmap_out_{i}.sam \
-p 6



    


















    






BSMAP v2.74
Start at:  Tue Oct 11 08:02:27 2016

Input reference file: ../data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa 	(format: FASTA)
Load in 7658 db seqs, total size 557717710 bp. 8 secs passed
total_kmers: 43046721
Create seed table. 24 secs passed
max number of mismatches: read_length * 8% 	max gap size: 0
kmer cut-off ratio: 5e-07
max multi-hits: 100	max Ns: 5	seed size: 16	index interval: 4
quality cutoff: 0	base quality char: '!'
min fragment size:28	max fragemt size:500
start from read #1	end at read #4294967295
additional alignment: T in reads => C in reference
mapping strand: ++,-+
Single-end alignment(6 threads)
Input read file: M2.fastq.gz 	(format: gzipped FASTQ)
Output file: bsmap_out_M2.sam	 (format: SAM)
Thread #1: 	100000 reads finished. 30 secs passed
Thread #0: 	50000 reads finished. 30 secs passed
Thread #2: 	150000 reads finished. 31 secs passed
Thread #3: 	200000 reads finished. 31 secs passed
Thread #5: 	250000 reads finished. 31 secs passed
Thread #4: 	300000 reads finished. 31 secs passed
Thread #1: 	350000 reads finished. 36 secs passed
Thread #0: 	400000 reads finished. 36 secs passed
Thread #2: 	450000 reads finished. 36 secs passed
Thread #3: 	500000 reads finished. 36 secs passed
Thread #5: 	550000 reads finished. 37 secs passed
Thread #4: 	600000 reads finished. 37 secs passed
Thread #1: 	650000 reads finished. 42 secs passed
Thread #2: 	750000 reads finished. 42 secs passed
Thread #0: 	700000 reads finished. 42 secs passed
Thread #3: 	800000 reads finished. 42 secs passed
Thread #5: 	850000 reads finished. 42 secs passed
Thread #4: 	900000 reads finished. 43 secs passed
Thread #1: 	950000 reads finished. 48 secs passed
Thread #2: 	1000000 reads finished. 48 secs passed
Thread #3: 	1100000 reads finished. 48 secs passed
Thread #0: 	1050000 reads finished. 49 secs passed
Thread #5: 	1150000 reads finished. 49 secs passed
Thread #4: 	1200000 reads finished. 49 secs passed
Thread #1: 	1250000 reads finished. 54 secs passed
Thread #2: 	1300000 reads finished. 54 secs passed
Thread #3: 	1350000 reads finished. 55 secs passed
Thread #5: 	1450000 reads finished. 55 secs passed
Thread #4: 	1500000 reads finished. 55 secs passed
Thread #0: 	1400000 reads finished. 55 secs passed
Thread #1: 	1550000 reads finished. 60 secs passed
Thread #2: 	1600000 reads finished. 60 secs passed
Thread #3: 	1650000 reads finished. 61 secs passed
Thread #4: 	1750000 reads finished. 61 secs passed
Thread #5: 	1700000 reads finished. 61 secs passed
Thread #0: 	1800000 reads finished. 61 secs passed
Thread #1: 	1850000 reads finished. 67 secs passed
Thread #2: 	1900000 reads finished. 67 secs passed
Thread #3: 	1950000 reads finished. 68 secs passed
Thread #4: 	2000000 reads finished. 68 secs passed
Thread #5: 	2050000 reads finished. 68 secs passed
Thread #0: 	2100000 reads finished. 68 secs passed
Thread #1: 	2150000 reads finished. 73 secs passed
Thread #2: 	2200000 reads finished. 74 secs passed
Thread #3: 	2250000 reads finished. 74 secs passed
Thread #4: 	2300000 reads finished. 74 secs passed
Thread #5: 	2350000 reads finished. 74 secs passed
Thread #0: 	2400000 reads finished. 75 secs passed
Thread #1: 	2450000 reads finished. 80 secs passed
Thread #2: 	2500000 reads finished. 80 secs passed
Thread #3: 	2550000 reads finished. 80 secs passed
Thread #4: 	2600000 reads finished. 81 secs passed
Thread #5: 	2650000 reads finished. 81 secs passed
Thread #0: 	2700000 reads finished. 81 secs passed
Thread #2: 	2800000 reads finished. 86 secs passed
Thread #1: 	2750000 reads finished. 86 secs passed
Thread #3: 	2850000 reads finished. 86 secs passed
Thread #4: 	2900000 reads finished. 87 secs passed
Thread #5: 	2950000 reads finished. 87 secs passed
Thread #0: 	3000000 reads finished. 88 secs passed
Thread #2: 	3050000 reads finished. 92 secs passed
Thread #1: 	3100000 reads finished. 92 secs passed
Thread #3: 	3150000 reads finished. 92 secs passed
Thread #4: 	3200000 reads finished. 92 secs passed
Thread #5: 	3250000 reads finished. 93 secs passed
Thread #0: 	3300000 reads finished. 94 secs passed
Thread #2: 	3350000 reads finished. 98 secs passed
Thread #1: 	3400000 reads finished. 98 secs passed
Thread #3: 	3450000 reads finished. 98 secs passed
Thread #4: 	3500000 reads finished. 98 secs passed
Thread #5: 	3550000 reads finished. 99 secs passed
Thread #0: 	3600000 reads finished. 100 secs passed
Thread #2: 	3650000 reads finished. 104 secs passed
Thread #1: 	3700000 reads finished. 104 secs passed
Thread #3: 	3750000 reads finished. 104 secs passed
Thread #4: 	3800000 reads finished. 104 secs passed
Thread #5: 	3850000 reads finished. 105 secs passed
Thread #0: 	3900000 reads finished. 106 secs passed
Thread #2: 	3950000 reads finished. 110 secs passed
Thread #1: 	4000000 reads finished. 110 secs passed
Thread #3: 	4050000 reads finished. 110 secs passed
Thread #4: 	4100000 reads finished. 110 secs passed
Thread #5: 	4150000 reads finished. 111 secs passed
Thread #0: 	4200000 reads finished. 112 secs passed
Thread #2: 	4250000 reads finished. 116 secs passed
Thread #1: 	4300000 reads finished. 116 secs passed
Thread #3: 	4350000 reads finished. 116 secs passed
Thread #4: 	4400000 reads finished. 117 secs passed
Thread #5: 	4450000 reads finished. 117 secs passed
Thread #0: 	4500000 reads finished. 119 secs passed
Thread #2: 	4550000 reads finished. 122 secs passed
Thread #1: 	4600000 reads finished. 122 secs passed
Thread #3: 	4650000 reads finished. 122 secs passed
Thread #4: 	4700000 reads finished. 123 secs passed
Thread #5: 	4750000 reads finished. 123 secs passed
Thread #0: 	4800000 reads finished. 125 secs passed
Thread #2: 	4850000 reads finished. 128 secs passed
Thread #1: 	4900000 reads finished. 128 secs passed
Thread #3: 	4950000 reads finished. 129 secs passed
Thread #4: 	5000000 reads finished. 129 secs passed
Thread #5: 	5050000 reads finished. 129 secs passed
Thread #0: 	5100000 reads finished. 131 secs passed
Thread #2: 	5150000 reads finished. 134 secs passed
Thread #1: 	5200000 reads finished. 134 secs passed
Thread #3: 	5250000 reads finished. 134 secs passed
Thread #4: 	5300000 reads finished. 135 secs passed
Thread #5: 	5350000 reads finished. 135 secs passed
Thread #0: 	5400000 reads finished. 137 secs passed
Thread #2: 	5450000 reads finished. 140 secs passed
Thread #1: 	5500000 reads finished. 140 secs passed
Thread #3: 	5550000 reads finished. 141 secs passed
Thread #4: 	5600000 reads finished. 141 secs passed
Thread #5: 	5650000 reads finished. 141 secs passed
Thread #0: 	5700000 reads finished. 143 secs passed
Thread #2: 	5750000 reads finished. 147 secs passed
Thread #1: 	5800000 reads finished. 147 secs passed
Thread #3: 	5850000 reads finished. 147 secs passed
Thread #4: 	5900000 reads finished. 147 secs passed
Thread #5: 	5950000 reads finished. 148 secs passed
Thread #0: 	6000000 reads finished. 150 secs passed
Thread #2: 	6050000 reads finished. 153 secs passed
Thread #1: 	6100000 reads finished. 153 secs passed
Thread #3: 	6150000 reads finished. 153 secs passed
Thread #4: 	6200000 reads finished. 153 secs passed
Thread #5: 	6250000 reads finished. 154 secs passed
Thread #0: 	6300000 reads finished. 156 secs passed
Thread #1: 	6400000 reads finished. 160 secs passed
Thread #2: 	6350000 reads finished. 160 secs passed
Thread #4: 	6500000 reads finished. 160 secs passed
Thread #3: 	6450000 reads finished. 160 secs passed
Thread #5: 	6550000 reads finished. 161 secs passed
Thread #0: 	6600000 reads finished. 164 secs passed
Thread #1: 	6650000 reads finished. 166 secs passed
Thread #4: 	6750000 reads finished. 167 secs passed
Thread #2: 	6700000 reads finished. 167 secs passed
Thread #3: 	6800000 reads finished. 167 secs passed
Thread #5: 	6850000 reads finished. 168 secs passed
Thread #0: 	6900000 reads finished. 171 secs passed
Thread #1: 	6950000 reads finished. 173 secs passed
Thread #2: 	7050000 reads finished. 174 secs passed
Thread #4: 	7000000 reads finished. 174 secs passed
Thread #3: 	7100000 reads finished. 174 secs passed
Thread #5: 	7150000 reads finished. 174 secs passed
Thread #0: 	7200000 reads finished. 177 secs passed
Thread #1: 	7250000 reads finished. 179 secs passed
Thread #2: 	7300000 reads finished. 180 secs passed
Thread #4: 	7350000 reads finished. 180 secs passed
Thread #3: 	7400000 reads finished. 180 secs passed
Thread #5: 	7450000 reads finished. 180 secs passed
Thread #0: 	7500000 reads finished. 184 secs passed
Thread #1: 	7550000 reads finished. 186 secs passed
Thread #2: 	7600000 reads finished. 186 secs passed
Thread #4: 	7650000 reads finished. 187 secs passed
Thread #3: 	7700000 reads finished. 187 secs passed
Thread #5: 	7750000 reads finished. 187 secs passed
Thread #0: 	7800000 reads finished. 191 secs passed
Thread #1: 	7850000 reads finished. 193 secs passed
Thread #2: 	7900000 reads finished. 193 secs passed
Thread #4: 	7950000 reads finished. 193 secs passed
Thread #3: 	8000000 reads finished. 193 secs passed
Thread #5: 	8050000 reads finished. 193 secs passed
Thread #0: 	8100000 reads finished. 196 secs passed
Thread #1: 	8150000 reads finished. 198 secs passed
Thread #2: 	8200000 reads finished. 199 secs passed
Thread #4: 	8250000 reads finished. 199 secs passed
Thread #3: 	8300000 reads finished. 199 secs passed
Thread #5: 	8350000 reads finished. 199 secs passed
Thread #0: 	8400000 reads finished. 203 secs passed
Thread #1: 	8450000 reads finished. 205 secs passed
Thread #2: 	8500000 reads finished. 205 secs passed
Thread #4: 	8550000 reads finished. 205 secs passed
Thread #5: 	8650000 reads finished. 205 secs passed
Thread #3: 	8600000 reads finished. 205 secs passed
Thread #0: 	8700000 reads finished. 209 secs passed
Thread #1: 	8750000 reads finished. 210 secs passed
Thread #2: 	8800000 reads finished. 211 secs passed
Thread #4: 	8850000 reads finished. 211 secs passed
Thread #5: 	8900000 reads finished. 211 secs passed
Thread #3: 	8950000 reads finished. 211 secs passed
Thread #0: 	9000000 reads finished. 215 secs passed
Thread #1: 	9050000 reads finished. 216 secs passed
Thread #2: 	9100000 reads finished. 217 secs passed
Thread #4: 	9150000 reads finished. 217 secs passed
Thread #5: 	9200000 reads finished. 217 secs passed
Thread #3: 	9250000 reads finished. 217 secs passed
Thread #0: 	9300000 reads finished. 221 secs passed
Thread #1: 	9350000 reads finished. 222 secs passed
Thread #2: 	9400000 reads finished. 223 secs passed
Thread #4: 	9450000 reads finished. 223 secs passed
Thread #5: 	9500000 reads finished. 223 secs passed
Thread #3: 	9550000 reads finished. 223 secs passed
Thread #0: 	9600000 reads finished. 227 secs passed
Thread #1: 	9650000 reads finished. 228 secs passed
Thread #2: 	9700000 reads finished. 228 secs passed
Thread #4: 	9750000 reads finished. 229 secs passed
Thread #5: 	9800000 reads finished. 229 secs passed
Thread #3: 	9850000 reads finished. 229 secs passed
Thread #0: 	9900000 reads finished. 233 secs passed
Thread #1: 	9950000 reads finished. 234 secs passed
Thread #2: 	10000000 reads finished. 235 secs passed
Thread #4: 	10050000 reads finished. 235 secs passed
Thread #5: 	10100000 reads finished. 235 secs passed
Thread #3: 	10150000 reads finished. 235 secs passed
Thread #0: 	10200000 reads finished. 239 secs passed
Thread #1: 	10250000 reads finished. 240 secs passed
Thread #2: 	10300000 reads finished. 241 secs passed
Thread #4: 	10350000 reads finished. 241 secs passed
Thread #5: 	10400000 reads finished. 241 secs passed
Thread #3: 	10450000 reads finished. 241 secs passed
Thread #2: 	10564512 reads finished. 242 secs passed
Thread #0: 	10500000 reads finished. 244 secs passed
Thread #1: 	10550000 reads finished. 244 secs passed
Total number of aligned reads: 7966031 (75%)
Done.
Finished at Tue Oct 11 08:06:32 2016
Total time consumed:  245 secs

BSMAP v2.74
Start at:  Tue Oct 11 08:06:32 2016

Input reference file: ../data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa 	(format: FASTA)
Load in 7658 db seqs, total size 557717710 bp. 6 secs passed
total_kmers: 43046721
Create seed table. 22 secs passed
max number of mismatches: read_length * 8% 	max gap size: 0
kmer cut-off ratio: 5e-07
max multi-hits: 100	max Ns: 5	seed size: 16	index interval: 4
quality cutoff: 0	base quality char: '!'
min fragment size:28	max fragemt size:500
start from read #1	end at read #4294967295
additional alignment: T in reads => C in reference
mapping strand: ++,-+
Single-end alignment(6 threads)
Input read file: M3.fastq.gz 	(format: gzipped FASTQ)
Output file: bsmap_out_M3.sam	 (format: SAM)
Thread #2: 	50000 reads finished. 28 secs passed
Thread #0: 	100000 reads finished. 28 secs passed
Thread #4: 	150000 reads finished. 28 secs passed
Thread #3: 	200000 reads finished. 28 secs passed
Thread #1: 	300000 reads finished. 28 secs passed
Thread #5: 	250000 reads finished. 28 secs passed
Thread #2: 	350000 reads finished. 33 secs passed
Thread #0: 	400000 reads finished. 33 secs passed
Thread #4: 	450000 reads finished. 34 secs passed
Thread #3: 	500000 reads finished. 34 secs passed
Thread #1: 	550000 reads finished. 34 secs passed
Thread #5: 	600000 reads finished. 34 secs passed
Thread #2: 	650000 reads finished. 39 secs passed
Thread #0: 	700000 reads finished. 39 secs passed
Thread #4: 	750000 reads finished. 39 secs passed
Thread #3: 	800000 reads finished. 39 secs passed
Thread #1: 	850000 reads finished. 39 secs passed
Thread #5: 	900000 reads finished. 40 secs passed
Thread #2: 	950000 reads finished. 45 secs passed
Thread #0: 	1000000 reads finished. 45 secs passed
Thread #4: 	1050000 reads finished. 45 secs passed
Thread #3: 	1100000 reads finished. 45 secs passed
Thread #1: 	1150000 reads finished. 45 secs passed
Thread #5: 	1200000 reads finished. 45 secs passed
Thread #2: 	1250000 reads finished. 50 secs passed
Thread #0: 	1300000 reads finished. 50 secs passed
Thread #4: 	1350000 reads finished. 51 secs passed
Thread #3: 	1400000 reads finished. 51 secs passed
Thread #1: 	1450000 reads finished. 51 secs passed
Thread #5: 	1500000 reads finished. 51 secs passed
Thread #2: 	1550000 reads finished. 56 secs passed
Thread #0: 	1600000 reads finished. 56 secs passed
Thread #4: 	1650000 reads finished. 57 secs passed
Thread #3: 	1700000 reads finished. 57 secs passed
Thread #1: 	1750000 reads finished. 57 secs passed
Thread #5: 	1800000 reads finished. 57 secs passed
Thread #2: 	1850000 reads finished. 62 secs passed
Thread #0: 	1900000 reads finished. 62 secs passed
Thread #4: 	1950000 reads finished. 62 secs passed
Thread #3: 	2000000 reads finished. 62 secs passed
Thread #1: 	2050000 reads finished. 63 secs passed
Thread #5: 	2100000 reads finished. 63 secs passed
Thread #2: 	2150000 reads finished. 68 secs passed
Thread #0: 	2200000 reads finished. 68 secs passed
Thread #4: 	2250000 reads finished. 68 secs passed
Thread #3: 	2300000 reads finished. 68 secs passed
Thread #1: 	2350000 reads finished. 68 secs passed
Thread #5: 	2400000 reads finished. 69 secs passed
Thread #2: 	2450000 reads finished. 73 secs passed
Thread #0: 	2500000 reads finished. 73 secs passed
Thread #4: 	2550000 reads finished. 74 secs passed
Thread #3: 	2600000 reads finished. 74 secs passed
Thread #1: 	2650000 reads finished. 74 secs passed
Thread #5: 	2700000 reads finished. 74 secs passed
Thread #2: 	2750000 reads finished. 79 secs passed
Thread #0: 	2800000 reads finished. 79 secs passed
Thread #4: 	2850000 reads finished. 79 secs passed
Thread #3: 	2900000 reads finished. 79 secs passed
Thread #1: 	2950000 reads finished. 80 secs passed
Thread #5: 	3000000 reads finished. 80 secs passed
Thread #2: 	3050000 reads finished. 85 secs passed
Thread #0: 	3100000 reads finished. 85 secs passed
Thread #4: 	3150000 reads finished. 85 secs passed
Thread #3: 	3200000 reads finished. 85 secs passed
Thread #1: 	3250000 reads finished. 85 secs passed
Thread #5: 	3300000 reads finished. 86 secs passed
Thread #2: 	3350000 reads finished. 90 secs passed
Thread #0: 	3400000 reads finished. 91 secs passed
Thread #4: 	3450000 reads finished. 91 secs passed
Thread #3: 	3500000 reads finished. 91 secs passed
Thread #1: 	3550000 reads finished. 91 secs passed
Thread #5: 	3600000 reads finished. 92 secs passed
Thread #2: 	3650000 reads finished. 96 secs passed
Thread #0: 	3700000 reads finished. 96 secs passed
Thread #4: 	3750000 reads finished. 97 secs passed
Thread #3: 	3800000 reads finished. 97 secs passed
Thread #1: 	3850000 reads finished. 97 secs passed
Thread #5: 	3900000 reads finished. 97 secs passed
Thread #2: 	3950000 reads finished. 102 secs passed
Thread #0: 	4000000 reads finished. 102 secs passed
Thread #4: 	4050000 reads finished. 103 secs passed
Thread #3: 	4100000 reads finished. 103 secs passed
Thread #1: 	4150000 reads finished. 103 secs passed
Thread #5: 	4200000 reads finished. 104 secs passed
Thread #2: 	4250000 reads finished. 108 secs passed
Thread #0: 	4300000 reads finished. 108 secs passed
Thread #4: 	4350000 reads finished. 108 secs passed
Thread #3: 	4400000 reads finished. 109 secs passed
Thread #1: 	4450000 reads finished. 109 secs passed
Thread #5: 	4500000 reads finished. 109 secs passed
Thread #2: 	4550000 reads finished. 114 secs passed
Thread #0: 	4600000 reads finished. 114 secs passed
Thread #4: 	4650000 reads finished. 115 secs passed
Thread #3: 	4700000 reads finished. 115 secs passed
Thread #1: 	4750000 reads finished. 115 secs passed
Thread #5: 	4800000 reads finished. 116 secs passed
Thread #2: 	4850000 reads finished. 120 secs passed
Thread #0: 	4900000 reads finished. 120 secs passed
Thread #4: 	4950000 reads finished. 121 secs passed
Thread #3: 	5000000 reads finished. 121 secs passed
Thread #1: 	5050000 reads finished. 121 secs passed
Thread #5: 	5100000 reads finished. 121 secs passed
Thread #2: 	5150000 reads finished. 125 secs passed
Thread #0: 	5200000 reads finished. 126 secs passed
Thread #4: 	5250000 reads finished. 126 secs passed
Thread #3: 	5300000 reads finished. 126 secs passed
Thread #1: 	5350000 reads finished. 126 secs passed
Thread #5: 	5400000 reads finished. 127 secs passed
Thread #2: 	5450000 reads finished. 132 secs passed
Thread #0: 	5500000 reads finished. 132 secs passed
Thread #4: 	5550000 reads finished. 132 secs passed
Thread #3: 	5600000 reads finished. 132 secs passed
Thread #1: 	5650000 reads finished. 133 secs passed
Thread #5: 	5700000 reads finished. 133 secs passed
Thread #2: 	5750000 reads finished. 138 secs passed
Thread #0: 	5800000 reads finished. 138 secs passed
Thread #4: 	5850000 reads finished. 139 secs passed
Thread #3: 	5900000 reads finished. 139 secs passed
Thread #1: 	5950000 reads finished. 139 secs passed
Thread #5: 	6000000 reads finished. 140 secs passed
Thread #2: 	6050000 reads finished. 145 secs passed
Thread #0: 	6100000 reads finished. 145 secs passed
Thread #4: 	6150000 reads finished. 146 secs passed
Thread #3: 	6200000 reads finished. 146 secs passed
Thread #1: 	6250000 reads finished. 146 secs passed
Thread #5: 	6300000 reads finished. 147 secs passed
Thread #2: 	6350000 reads finished. 151 secs passed
Thread #0: 	6400000 reads finished. 152 secs passed
Thread #4: 	6450000 reads finished. 152 secs passed
Thread #3: 	6500000 reads finished. 152 secs passed
Thread #1: 	6550000 reads finished. 152 secs passed
Thread #5: 	6600000 reads finished. 153 secs passed
Thread #2: 	6650000 reads finished. 158 secs passed
Thread #0: 	6700000 reads finished. 159 secs passed
Thread #4: 	6750000 reads finished. 159 secs passed
Thread #3: 	6800000 reads finished. 159 secs passed
Thread #1: 	6850000 reads finished. 159 secs passed
Thread #5: 	6900000 reads finished. 160 secs passed
Thread #2: 	6950000 reads finished. 165 secs passed
Thread #0: 	7000000 reads finished. 166 secs passed
Thread #4: 	7050000 reads finished. 166 secs passed
Thread #1: 	7150000 reads finished. 166 secs passed
Thread #3: 	7100000 reads finished. 166 secs passed
Thread #5: 	7200000 reads finished. 167 secs passed
Thread #2: 	7250000 reads finished. 171 secs passed
Thread #0: 	7300000 reads finished. 172 secs passed
Thread #4: 	7350000 reads finished. 172 secs passed
Thread #3: 	7450000 reads finished. 172 secs passed
Thread #1: 	7400000 reads finished. 173 secs passed
Thread #5: 	7500000 reads finished. 173 secs passed
Thread #2: 	7550000 reads finished. 178 secs passed
Thread #0: 	7600000 reads finished. 178 secs passed
Thread #4: 	7650000 reads finished. 178 secs passed
Thread #3: 	7700000 reads finished. 179 secs passed
Thread #1: 	7750000 reads finished. 179 secs passed
Thread #5: 	7800000 reads finished. 179 secs passed
Thread #2: 	7850000 reads finished. 184 secs passed
Thread #0: 	7900000 reads finished. 184 secs passed
Thread #4: 	7950000 reads finished. 184 secs passed
Thread #3: 	8000000 reads finished. 184 secs passed
Thread #1: 	8050000 reads finished. 185 secs passed
Thread #5: 	8100000 reads finished. 185 secs passed
Thread #2: 	8150000 reads finished. 189 secs passed
Thread #0: 	8200000 reads finished. 190 secs passed
Thread #4: 	8250000 reads finished. 190 secs passed
Thread #3: 	8300000 reads finished. 190 secs passed
Thread #1: 	8350000 reads finished. 191 secs passed
Thread #5: 	8400000 reads finished. 191 secs passed
Thread #2: 	8450000 reads finished. 195 secs passed
Thread #0: 	8500000 reads finished. 196 secs passed
Thread #4: 	8550000 reads finished. 196 secs passed
Thread #3: 	8600000 reads finished. 196 secs passed
Thread #1: 	8650000 reads finished. 197 secs passed
Thread #5: 	8700000 reads finished. 197 secs passed
Thread #2: 	8750000 reads finished. 202 secs passed
Thread #0: 	8800000 reads finished. 202 secs passed
Thread #4: 	8850000 reads finished. 202 secs passed
Thread #3: 	8900000 reads finished. 202 secs passed
Thread #1: 	8950000 reads finished. 203 secs passed
Thread #5: 	9000000 reads finished. 203 secs passed
Thread #2: 	9050000 reads finished. 208 secs passed
Thread #0: 	9100000 reads finished. 208 secs passed
Thread #4: 	9150000 reads finished. 208 secs passed
Thread #3: 	9200000 reads finished. 209 secs passed
Thread #1: 	9250000 reads finished. 209 secs passed
Thread #5: 	9300000 reads finished. 209 secs passed
Thread #2: 	9350000 reads finished. 214 secs passed
Thread #4: 	9450000 reads finished. 214 secs passed
Thread #0: 	9400000 reads finished. 214 secs passed
Thread #3: 	9500000 reads finished. 215 secs passed
Thread #1: 	9550000 reads finished. 215 secs passed
Thread #5: 	9600000 reads finished. 215 secs passed
Thread #2: 	9650000 reads finished. 220 secs passed
Thread #4: 	9700000 reads finished. 220 secs passed
Thread #0: 	9750000 reads finished. 220 secs passed
Thread #3: 	9800000 reads finished. 220 secs passed
Thread #1: 	9850000 reads finished. 220 secs passed
Thread #5: 	9900000 reads finished. 221 secs passed
Thread #2: 	9950000 reads finished. 226 secs passed
Thread #4: 	10000000 reads finished. 226 secs passed
Thread #0: 	10050000 reads finished. 226 secs passed
Thread #3: 	10100000 reads finished. 226 secs passed
Thread #1: 	10150000 reads finished. 226 secs passed
Thread #5: 	10200000 reads finished. 227 secs passed
Thread #2: 	10250000 reads finished. 231 secs passed
Thread #4: 	10300000 reads finished. 232 secs passed
Thread #0: 	10350000 reads finished. 232 secs passed
Thread #3: 	10400000 reads finished. 232 secs passed
Thread #1: 	10450000 reads finished. 232 secs passed
Thread #5: 	10500000 reads finished. 233 secs passed
Thread #2: 	10550000 reads finished. 238 secs passed
Thread #4: 	10600000 reads finished. 238 secs passed
Thread #0: 	10650000 reads finished. 238 secs passed
Thread #3: 	10700000 reads finished. 238 secs passed
Thread #1: 	10750000 reads finished. 238 secs passed
Thread #5: 	10800000 reads finished. 239 secs passed
Thread #2: 	10850000 reads finished. 244 secs passed
Thread #4: 	10900000 reads finished. 244 secs passed
Thread #0: 	10950000 reads finished. 244 secs passed
Thread #3: 	11000000 reads finished. 244 secs passed
Thread #1: 	11050000 reads finished. 244 secs passed
Thread #5: 	11100000 reads finished. 245 secs passed
Thread #2: 	11150000 reads finished. 249 secs passed
Thread #4: 	11200000 reads finished. 249 secs passed
Thread #0: 	11250000 reads finished. 250 secs passed
Thread #3: 	11300000 reads finished. 250 secs passed
Thread #1: 	11350000 reads finished. 250 secs passed
Thread #5: 	11400000 reads finished. 250 secs passed
Thread #5: 	11685119 reads finished. 255 secs passed
Thread #2: 	11450000 reads finished. 255 secs passed
Thread #4: 	11500000 reads finished. 255 secs passed
Thread #0: 	11550000 reads finished. 255 secs passed
Thread #3: 	11600000 reads finished. 256 secs passed
Thread #1: 	11650000 reads finished. 256 secs passed
Total number of aligned reads: 8721542 (75%)
Done.
Finished at Tue Oct 11 08:10:48 2016
Total time consumed:  256 secs

















In [42]:



    


for i in ("M2","M3"):
    !python {bsmaploc}methratio.py \
-d ../data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa \
-u -z -g \
-o methratio_out_{i}.txt \
-s {bsmaploc}samtools \
bsmap_out_{i}.sam \



    


















    






@ Tue Oct 11 08:10:48 2016: reading reference ../data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa ...
@ Tue Oct 11 08:11:11 2016: reading bsmap_out_M2.sam ...
[samopen] SAM header is present: 7658 sequences.
@ Tue Oct 11 08:13:23 2016: combining CpG methylation from both strands ...
@ Tue Oct 11 08:13:44 2016: writing methratio_out_M2.txt ...
@ Tue Oct 11 08:16:58 2016: done.
total 5453853 valid mappings, 21735799 covered cytosines, average coverage: 1.86 fold.
@ Tue Oct 11 08:16:59 2016: reading reference ../data/Crassostrea_gigas.GCAz_000297895.1.dna_sm.toplevel.fa ...
@ Tue Oct 11 08:17:21 2016: reading bsmap_out_M3.sam ...
[samopen] SAM header is present: 7658 sequences.
@ Tue Oct 11 08:19:41 2016: combining CpG methylation from both strands ...
@ Tue Oct 11 08:20:00 2016: writing methratio_out_M3.txt ...
@ Tue Oct 11 08:23:17 2016: done.
total 5974521 valid mappings, 23104419 covered cytosines, average coverage: 1.89 fold.

















In [43]:



    


!curl https://raw.githubusercontent.com/che625/olson-ms-nb/master/scripts/mr3x.awk \
> /Users/sr320/git-repos/sr320.github.io/jupyter/scripts/mr3x.awk

!curl https://raw.githubusercontent.com/che625/olson-ms-nb/master/scripts/mr_gg.awk.sh \
> /Users/sr320/git-repos/sr320.github.io/jupyter/scripts/mr_gg.awk.sh



    


















    






  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    83  100    83    0     0     88      0 --:--:-- --:--:-- --:--:--    88
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   286  100   286    0     0   1665      0 --:--:-- --:--:-- --:--:--  1672

















In [45]:



    


#first methratio files are converted to filter for CG context, 3x coverage (mr3x.awk), and reformatting (mr_gg.awk.sh).
#due to issue passing variable to awk, simple scripts were used (included in repository)
for i in ("M2","M3"):
    !echo {i}
    !grep "[A-Z][A-Z]CG[A-Z]" <methratio_out_{i}.txt> methratio_out_{i}CG.txt
    !awk -f /Users/sr320/git-repos/sr320.github.io/jupyter/scripts/mr3x.awk methratio_out_{i}CG.txt \
    > mr3x.{i}.txt
    !awk -f /Users/sr320/git-repos/sr320.github.io/jupyter/scripts/mr_gg.awk.sh \
    mr3x.{i}.txt > mkfmt_{i}.txt



    


















    






M2
M3

















In [47]:



    


#first methratio files are converted to filter for CG context, 3x coverage (mr3x.awk), and reformatting (mr_gg.awk.sh).
#due to issue passing variable to awk, simple scripts were used (included in repository)
for i in ("M2","M3"):
    !echo {i}
    !grep -i "[A-Z][A-Z]CG[A-Z]" <methratio_out_{i}.txt> methratio_out_{i}CGi.txt
    !awk -f /Users/sr320/git-repos/sr320.github.io/jupyter/scripts/mr3x.awk methratio_out_{i}CGi.txt \
    > mr3xi.{i}.txt
    !awk -f /Users/sr320/git-repos/sr320.github.io/jupyter/scripts/mr_gg.awk.sh \
    mr3xi.{i}.txt > mkfmti_{i}.txt



    


















    






M2
M3

















In [46]:



    


#maybe we need to ignore case



    











In [51]:



    


!md5 mkfmt_M2.txt mkfmti_M2.txt | head



    


















    






MD5 (mkfmt_M2.txt) = df67fde9e87ec165618d384374074057
MD5 (mkfmti_M2.txt) = df67fde9e87ec165618d384374074057

















In [52]:



    


#nope



    











In [54]:



    


!head -100 mkfmt_M2.txt



    


















    






chr.Base	chr	base	strand	coverage	freqC	freqT
scaffold1.14274	scaffold1	14274	F	4	0.00	100.00
scaffold1.14305	scaffold1	14305	F	4	0.00	100.00
scaffold1.15309	scaffold1	15309	F	4	0.00	100.00
scaffold1.15315	scaffold1	15315	F	4	0.00	100.00
scaffold1.15336	scaffold1	15336	F	5	0.00	100.00
scaffold1.16016	scaffold1	16016	F	3	0.00	100.00
scaffold1.17209	scaffold1	17209	F	3	0.00	100.00
scaffold1.19038	scaffold1	19038	F	4	0.00	100.00
scaffold1.19069	scaffold1	19069	F	4	0.00	100.00
scaffold1.19077	scaffold1	19077	F	3	0.00	100.00
scaffold1.19147	scaffold1	19147	F	7	0.00	100.00
scaffold1.19211	scaffold1	19211	F	3	0.00	100.00
scaffold1.19222	scaffold1	19222	F	3	0.00	100.00
scaffold1.19427	scaffold1	19427	F	5	0.00	100.00
scaffold1.19454	scaffold1	19454	F	3	0.00	100.00
scaffold1.19456	scaffold1	19456	F	3	0.00	100.00
scaffold1.19864	scaffold1	19864	F	3	0.00	100.00
scaffold1.19935	scaffold1	19935	F	5	0.00	100.00
scaffold1.19963	scaffold1	19963	F	4	0.00	100.00
scaffold1.20056	scaffold1	20056	F	7	0.00	100.00
scaffold1.20067	scaffold1	20067	F	7	0.00	100.00
scaffold1.20087	scaffold1	20087	F	5	0.00	100.00
scaffold1.20116	scaffold1	20116	F	3	0.00	100.00
scaffold1.20119	scaffold1	20119	F	3	0.00	100.00
scaffold1.20180	scaffold1	20180	F	4	0.00	100.00
scaffold1.20215	scaffold1	20215	F	5	0.00	100.00
scaffold1.20253	scaffold1	20253	F	6	0.00	100.00
scaffold1.20275	scaffold1	20275	F	5	0.00	100.00
scaffold1.20278	scaffold1	20278	F	5	0.00	100.00
scaffold1.20560	scaffold1	20560	F	3	0.00	100.00
scaffold1.20564	scaffold1	20564	F	3	0.00	100.00
scaffold1.20756	scaffold1	20756	F	4	25.00	75.00
scaffold1.21325	scaffold1	21325	F	3	33.33	66.67
scaffold1.21633	scaffold1	21633	F	4	25.00	75.00
scaffold1.22985	scaffold1	22985	F	3	100.00	0.00
scaffold1.23091	scaffold1	23091	F	3	33.33	66.67
scaffold1.23095	scaffold1	23095	F	5	40.00	60.00
scaffold1.23099	scaffold1	23099	F	3	66.67	33.33
scaffold1.23469	scaffold1	23469	F	5	0.00	100.00
scaffold1.23979	scaffold1	23979	F	5	80.00	20.00
scaffold1.24040	scaffold1	24040	F	3	66.67	33.33
scaffold1.24627	scaffold1	24627	F	3	0.00	100.00
scaffold1.25289	scaffold1	25289	F	12	83.33	16.67
scaffold1.25324	scaffold1	25324	F	8	50.00	50.00
scaffold1.25489	scaffold1	25489	F	6	16.67	83.33
scaffold1.25503	scaffold1	25503	F	3	100.00	0.00
scaffold1.25517	scaffold1	25517	F	3	0.00	100.00
scaffold1.30312	scaffold1	30312	F	3	0.00	100.00
scaffold1.30364	scaffold1	30364	F	5	0.00	100.00
scaffold1.34376	scaffold1	34376	F	3	0.00	100.00
scaffold1.34387	scaffold1	34387	F	8	12.50	87.50
scaffold1.34409	scaffold1	34409	F	7	0.00	100.00
scaffold1.37888	scaffold1	37888	F	3	0.00	100.00
scaffold1.38052	scaffold1	38052	F	3	0.00	100.00
scaffold1.38200	scaffold1	38200	F	3	0.00	100.00
scaffold1.38307	scaffold1	38307	F	3	0.00	100.00
scaffold1.38396	scaffold1	38396	F	3	0.00	100.00
scaffold1.38420	scaffold1	38420	F	8	0.00	100.00
scaffold1.38451	scaffold1	38451	F	6	0.00	100.00
scaffold1.38454	scaffold1	38454	F	6	0.00	100.00
scaffold1.38457	scaffold1	38457	F	7	0.00	100.00
scaffold1.38473	scaffold1	38473	F	7	0.00	100.00
scaffold1.38477	scaffold1	38477	F	7	14.29	85.71
scaffold1.38502	scaffold1	38502	F	8	0.00	100.00
scaffold1.38528	scaffold1	38528	F	4	0.00	100.00
scaffold1.41897	scaffold1	41897	F	3	0.00	100.00
scaffold1.42178	scaffold1	42178	F	6	0.00	100.00
scaffold1.42189	scaffold1	42189	F	4	0.00	100.00
scaffold1.42210	scaffold1	42210	F	6	0.00	100.00
scaffold1.43637	scaffold1	43637	F	4	0.00	100.00
scaffold1.43648	scaffold1	43648	F	4	0.00	100.00
scaffold1.43788	scaffold1	43788	F	3	0.00	100.00
scaffold1.43818	scaffold1	43818	F	3	0.00	100.00
scaffold1.44516	scaffold1	44516	F	3	0.00	100.00
scaffold1.44713	scaffold1	44713	F	5	0.00	100.00
scaffold1.44719	scaffold1	44719	F	6	0.00	100.00
scaffold1.44860	scaffold1	44860	F	3	0.00	100.00
scaffold1.44862	scaffold1	44862	F	3	0.00	100.00
scaffold1.45677	scaffold1	45677	F	3	0.00	100.00
scaffold1.45683	scaffold1	45683	F	3	0.00	100.00
scaffold1.45736	scaffold1	45736	F	5	0.00	100.00
scaffold1.47244	scaffold1	47244	F	8	0.00	100.00
scaffold1.47265	scaffold1	47265	F	5	20.00	80.00
scaffold1.49801	scaffold1	49801	F	4	0.00	100.00
scaffold1.50021	scaffold1	50021	F	5	0.00	100.00
scaffold1.52362	scaffold1	52362	F	5	0.00	100.00
scaffold1.52371	scaffold1	52371	F	5	0.00	100.00
scaffold1.52400	scaffold1	52400	F	3	0.00	100.00
scaffold1.52406	scaffold1	52406	F	3	0.00	100.00
scaffold1.52408	scaffold1	52408	F	4	0.00	100.00
scaffold1.52418	scaffold1	52418	F	5	0.00	100.00
scaffold1.52422	scaffold1	52422	F	4	0.00	100.00
scaffold1.52454	scaffold1	52454	F	4	0.00	100.00
scaffold1.52467	scaffold1	52467	F	3	0.00	100.00
scaffold1.52472	scaffold1	52472	F	3	0.00	100.00
scaffold1.52512	scaffold1	52512	F	4	0.00	100.00
scaffold1.52528	scaffold1	52528	F	4	0.00	100.00
scaffold1.53044	scaffold1	53044	F	5	0.00	100.00
scaffold1.53050	scaffold1	53050	F	5	0.00	100.00

Products





In [56]:



    


cd /Users/sr320/git-repos/sr320.github.io/jupyter



    


















    






/Users/sr320/git-repos/sr320.github.io/jupyter

















In [58]:



    


mkdir analyses



    











In [59]:



    


mkdir analyses/$(date +%F)



    











In [61]:



    


for i in ("M2","M3"):
    !cp /Volumes/caviar/wd/2016-10-11/mkfmt_{i}.txt analyses/$(date +%F)/mkfmt_{i}.txt



    











In [63]:



    


!head analyses/$(date +%F)/*



    


















    






==> analyses/2016-10-11/mkfmt_M2.txt <==
chr.Base	chr	base	strand	coverage	freqC	freqT
scaffold1.14274	scaffold1	14274	F	4	0.00	100.00
scaffold1.14305	scaffold1	14305	F	4	0.00	100.00
scaffold1.15309	scaffold1	15309	F	4	0.00	100.00
scaffold1.15315	scaffold1	15315	F	4	0.00	100.00
scaffold1.15336	scaffold1	15336	F	5	0.00	100.00
scaffold1.16016	scaffold1	16016	F	3	0.00	100.00
scaffold1.17209	scaffold1	17209	F	3	0.00	100.00
scaffold1.19038	scaffold1	19038	F	4	0.00	100.00
scaffold1.19069	scaffold1	19069	F	4	0.00	100.00

==> analyses/2016-10-11/mkfmt_M3.txt <==
chr.Base	chr	base	strand	coverage	freqC	freqT
scaffold1.259	scaffold1	259	F	4	100.00	0.00
scaffold1.263	scaffold1	263	F	4	100.00	0.00
scaffold1.267	scaffold1	267	F	4	100.00	0.00
scaffold1.271	scaffold1	271	F	4	100.00	0.00
scaffold1.299	scaffold1	299	F	4	50.00	50.00
scaffold1.10033	scaffold1	10033	F	3	0.00	100.00
scaffold1.10343	scaffold1	10343	F	4	0.00	100.00
scaffold1.10530	scaffold1	10530	F	4	0.00	100.00
scaffold1.10569	scaffold1	10569	F	5	0.00	100.00

urls

https://raw.githubusercontent.com/sr320/sr320.github.io/master/jupyter/analyses/2016-10-11/mkfmt_M2.txt

https://raw.githubusercontent.com/sr320/sr320.github.io/master/jupyter/analyses/2016-10-11/mkfmt_M3.txt




In [ ]:

Content source: sr320/sr320.github.io

Similar notebooks:

notebook.community | gallery | about